Goal-driven authoring assistance using causal stylistic prescriptions

ABSTRACT

A method for generating stylistic feature prescriptions to align a body of text with one or more target goals includes receiving, at a stylistic feature model, a body of text, where the body of text is selected by a user via a graphical user interface (GUI). The stylistic feature model identifies stylistic features from the body of text and populates a stylistic feature vector with the stylistic features. A trained de-confounded prediction model receives the stylistic feature vector. The trained de-confounded prediction model using the stylistic feature vector generates a prediction value for each of one or more target goals, compares the prediction value for each of the one or more target goals to a target value for each of the one or more target goals and outputs, for display on the GUI, one or more stylistic feature prescriptions to the body of text based on results of the comparing.

TECHNICAL FIELD

This description relates to goal-driven authoring assistance usingcausal stylistic prescriptions.

BACKGROUND

Enterprise content authors write a piece of content to achieve a certaintarget goal, for example, to achieve higher shareability of content. Thetarget goal can range from improving content consumption, achievingbrand conformance, shareability on social media, search-engineoptimization for discoverability, improving reading time,comprehensibility, or achieving a target affect or tone (e.g., valence,arousal, dominance) with the reader. Often these target goals are whatan enterprise, or an author might want to achieve with the help of thecontent. Each of these goals would require the author to adopt differentstyle of writing. In these scenarios, authors often look for guidingprescriptions that help them in paraphrasing content to better achievethe target goal. The authors can use such prescriptions to stylisticallytune their content to achieve the goal.

For instance, consider the formality in content as a target goal.Intuitively, there are several characteristics that may be used toidentify the formality of a piece of text, namely: (i) the manner inwhich punctuations are used (surface-level style elements), (ii) choiceof words—more objective words than subjective words; ‘residence’ insteadof ‘home’ (lexical-level style elements), and (iii) the way sentencesare structured—more syntactically complex sentences than sentences withsimple syntax (syntax-level style elements). Thus, formality in text isexpressed at various levels, namely, surface, lexical, and syntactic.While these levels of style influencing formality, and the elementswithin them, are known intuitively, a similar understanding might not beavailable for several other target goals such as, for example,comprehensibility, shareability, discoverability, consumption, etc.However, having such an intuition at different levels of style can helpauthors improve their content style to achieve the target goal.

Given the non-trivial extension of stylistic intuitions to sophisticatedtarget-strategies, and a multitude of such target goals in authoring, itis not possible to provide accurate and useful prescriptions fortailoring content towards a goal using a rule-based approach. On theother hand, a conventional function learning approach would alsoestimate the effect of stylistic elements on target goal, but this willbe confounded by latent variables. This calls for a more principledapproach for providing such prescriptions.

Within industry in the author assistance space, some focus on the needsof enterprises and some focus on the needs of independent contentcreators (e.g., authors and editors, specifically). One approach thatfocuses on the needs of enterprises claims to capture all historicenterprise communications to first understand the implicit enterpriseguidelines from the content to provide assistance to content writers.This approach limits suggestions to spellings, grammatical errors, styleinconsistencies, use of deprecated terms, etc. While these prescriptionsmay help in creating content with consistent style aspects, in theabsence of a connection to business objectives/goals, they do not helpin achieving the enterprise business needs.

One approach that focuses on the needs of independent content creatorsuses authoring-time assistance features to facilitate individual contentwriters and has no notion of stylistic guidelines or norms that arespecific to an enterprise. Moreover, while politeness and formality canbe thought of as target strategies, they do not cover all potentialtarget strategies and extensions to include other target strategiesinvolving access to heavy label-specific, language-based resources,which are not easily available. There has been a growing interest insolving natural language tasks related to style in text. However, theseapproaches have been limited due to their assumptions about style andits composition. They use stylistic intuitions that are linked todifferences in style—be it genre classification, author profiling,social relationship classification, sentiment analysis, readabilityclassification, stylized text generation or style transfer. Theseassumptions are often task-specific and do not cover all aspects ofstyle leading to a need to fill the gap between the understanding ofstyle and solving tasks related to it. What is needed is a structuredway to develop a task-independent understanding of style—that is nottied to any of the above tasks, and is general enough to encompass them.

Other conventional efforts that revolve around style in text, arelacking primarily on two fronts: the focus is on identifying relationsbetween the content and the target goal. These explorations whenextended to prescriptive applications for authoring assistance areheavily resource dependent and often inhibitory in terms of cognitivespace available to the authors/editors. These established relationsbetween the content and target business strategies are also notcausal—therefore, they cannot be directly extended for prescriptions.Aligned with this trend, there has been an abundance of research thataims to leverage supervised learning approaches that range from simplelogistic regression models to more sophisticated deep neural networks inorder to establish correlations between the observed features of thecontent (x) and the target variables (y) that quantify target goals. Itis well known that causations cannot be inferred from correlations—thestrongly correlated features may not necessarily be the ones thatinfluence the target variables. Instead, they may indirectly do so byinfluencing an unobserved confounder z—a variable that influences both xand y. A need exists to align the process of content creation withtarget strategies, in a prescriptive setting.

SUMMARY

According to one general aspect, a computer-implemented method forgenerating stylistic feature prescriptions to align a body of text withone or more target goals includes receiving, at a stylistic featuremodel, a body of text, where the body of text is selected by a user viaa graphical user interface (GUI). The stylistic feature model identifiesstylistic features from the body of text and populates a stylisticfeature vector with the stylistic features. A trained de-confoundedprediction model receives the stylistic feature vector. The trainedde-confounded prediction model using the stylistic feature vectorgenerates a prediction value for each of one or more target goals,compares the prediction value for each of the one or more target goalsto a target value for each of the one or more target goals and outputs,for display on the GUI, one or more stylistic feature prescriptions tothe body of text based on results of the comparing. The above method maybe implemented as a computer-implemented method and as a computerprogram product.

In another general aspect, a system for generating stylistic featureprescriptions to align a body of text with one or more target goalsincludes at least one memory including instructions and at least oneprocessor that is operably coupled to the at least one memory and thatis arranged and configured to execute instructions that, when executed,cause the at least one processor to implement an application. Theapplication includes a graphical user interface (GUI) configured toindicate a selection of a body of text. The application includes astylistic feature model configured to receive the body of text, identifystylistic features from the body of text, and populate a stylisticfeature vector with the stylistic features. The application includes atrained de-confounded prediction model that is configured to receive thestylistic feature vector, generate a prediction value for each of theone or more target goals using the stylistic feature vector, compare theprediction value for each of the one or more target goals to a targetvalue for each of the one or more target goals, and output one or morestylistic feature prescriptions to the body of text based on results ofthe comparing. The GUI is configured to display the one or morestylistic feature prescriptions to the body of text.

The details of one or more implementations are set forth in theaccompa-nying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example chart for a stylistic feature structure.

FIG. 2 is an example chart for a stylistic feature structure and targetgoals.

FIG. 3 is a block diagram of an algorithm to model target goals fromstylistic features.

FIG. 4 is a block diagram of the de-confounded prediction model fromFIG. 3.

FIG. 5 is a block diagram of an algorithm to prescribe stylisticfeatures to a received body of text.

FIG. 6 is a block diagram of an algorithm for difference modeling.

FIG. 7 is an example graphical user interface illustrating prescribedstylistic features for a body of text.

FIG. 8 is an example block diagram of a system for generating stylisticfeature prescriptions.

DETAILED DESCRIPTION

This document describes a technical solution to the above-describedshortcoming by providing a causal modeling-based approach to provideprescriptions to authors related to various stylistic aspects of contenttowards achieving a target goal. Prescriptions, as used herein, aresuggestions made to the author for ways to change various stylisticaspects of their content to achieve a target goal. The prescriptions arebased on the estimation of causal effects of various stylistic elements(surface, lexical, syntactic) towards the target outcome (i.e., thetarget goal). By incorporating the prescriptions into the content andchanging the content by following the suggestions, the authors can thuscompose (or paraphrase) their content in a manner that better alignswith their goals. Such an assistance will also reduce the cognitive loadon authors/editors and will provide them with necessary signals to tunetheir content in the right direction.

In general, given a choice of target goal and its desired value alongwith the input content (e.g., a body of text), the technical solutionuses a structured identification and quantification of stylisticelements in text—to qualify the style at surface-level, lexical-level,and syntactic-level. A trained de-confounded prediction model thenprescribes changes to different style features of the input content,where the trained de-confounded prediction model has been trained usingcontent that has successfully achieved the target goal in the past. Forexample, given shareability as a target goal, the algorithm using thetrained de-confounded prediction model might suggest the best style touse of exclamations, subjective words, and simple sentences if they haveshown success in past content. In this manner, a structuredidentification and quantification of elements of style is developed andused to train a de-confounded prediction model, which is a causal modelof the target goals with respect to these identified style features.Then, the trained de-confounded prediction model is used to providestylistic feature prescriptions in real-time for a body of text.

Given a choice of target goal and its desired value along with the inputcontent, the technical solution uses a structured identification andquantification of stylistic elements in text—to quantify the style at asurface-level, a lexical level, and a syntactic-level. The model thenprescribes changes to different style aspects of the input content basedon a causal model of the content that have successfully achieved thetarget goal in the past. For example, given shareability as a targetgoal, the algorithm in the technical solution might suggest the beststyle includes the use of exclamations, subjective words, and simplesentences if they have shown success in past content. Thus, thetechnical solution includes a structured identification andquantification of elements of style, causal modeling of the target goalswith respect to these identified style elements and utilization of thismodel to provide stylistic prescriptions in real-time.

Within industry, at least one traditional approach provides no real-timefeature that can help the author/editor in aligning a piece of text witha target goal that they (or their enterprise) may have in mind. Thistraditional approach highlights the absence of such features that enablestrategic tuning of text to match creators' requirements. The technicalsolution and approach described in this document addresses thisshortcoming of this traditional solution by a causal modeling of targetstrategies to provide style prescriptions that are specific to theenterprise objective/goal(s).

Within industry, at least one traditional approach provides suggestionsthat can help creators make the content more polite and formal. However,these suggestions are highly specific to the target variables and theunderlying methodology cannot be extended to encompass other targetstrategies (such as, shareability on social media, search-engineoptimization, reading time, comprehensibility, etc.). Further, thesuggestions are content-based and not style-based—and require a mappingof lexicons. Moreover, polite and formal cannot be identified onaffective dimensions. Politeness and formality have been identified asone of the many factors that bring about certain variations inshareability, discoverability, consumability, reading time, and affectin the reader (which can be identified as one of the target strategies).In this regard, the technical solution described in this documentbridges this gap by directly providing causal suggestions to tunecontent in order to meet the target strategy.

Additionally, at least one traditional approach operates at the level ofan individual creator—and hence, doesn't exploit previous publicationsby an enterprise to implicitly understand their stylistic norms andguidelines. On the other hand, the technical solution described in thisdocument leverages data-driven methods to implicitly modelcausal-relationships between (a) stylistic aspects that are prominent inprevious publications and (b) target strategy at hand, to recommendfine-tuning of content in order to match the strategy that thecreator/enterprise has in mind. The technical solution described in thisdocument does not rely on label-specific language-based resources andextends to multiple target strategies. Being a data-driven approach, itimplicitly captures the stylistic guidelines and norms that anenterprise adheres to.

Within academia, there has been a lack of clear structure for discussingand identifying stylistic aspects in text. The technical solutiondescribed in this document including the demarcation of stylisticaspects into surface, syntactic, and lexical levels facilitates suchacademic deliberation and identification.

While existing methods aim to establish relations between underlyingcontent of a text and the affect it evokes, the technical solutiondescribed in this document aims to establish relations between stylisticaspects of text and the target strategy. In other words, realizations(r) (i.e., sentences) can be dissociated into content (c) andstyle-related aspects (s). I.e., r=f (c, s). While existing works aim toestablish non-causal relations between c and strategy y, the technicalsolution described in this document establishes causal relations betweens and y. Leveraging the relations between c and y in a prescriptivesetting is highly dependent on availability exhaustive resources, whileleveraging relations between s and y is relatively less demanding.Consider the following example:

For providing a content-based stylistic suggestion (e.g., use the word“tasty” instead of “palatable” for inducing more excitement in thetext), one way is to have a mapping of words along the dimension ofexcitement—this would require plenty of resources—(n−1) mappings foreach word, given n affective dimensions along with a lexicon for each ofthose affective dimensions. However, for providing a style-basedsuggestion (e.g., use a more subjective words than objective words forinducing more excitement in the text) will require no such requirementof lexical mappings, but only a lexicon for each of the n affectivedimensions.

Along with this, style-based stylistic changes (as opposed tocontent-based stylistic changes) allows the author to understand thedeviations in content at a macroscopic level and hence, they have morecognitive space to align the content with the desired target strategy.In this sense, style-based stylistic prescriptions are less restrictiveas they provide more cognitive space.

The technical solution aligns the process of content creation withtarget goals, in a prescriptive setting. To this end, the technicalsolution overcomes the shortcomings identified above by aiming toestablish causal relations between stylistic aspects of content and thetarget business goal. The technical solution uses a quantification ofstyle elements at different levels along with causal inference toprovide stylistic prescription to achieve target goals.

First, a structure for the stylistic features is introduced. FIG. 1 isan example chart 100 for a stylistic feature structure. The chart 100provides a structured identification and quantification of stylisticfeatures (also referred to interchangeably as stylistic elements) in abody of text. The structure is used to identify and quantify thestylistic features from a given piece of text. In some implementations,the structured identification and quantification of stylistic featuresincludes stylistic features at three different levels—surface-level 102(also referred to as surface), lexical-level 104 (also referred to aslexical), and syntactic-level 106 (also referred to as syntactic). Thisstructure is first used to train a de-confounded prediction model. Then,the trained de-confounded prediction model is used to provideprescriptions (or suggestions) to a user for how to change an input bodyof text to meet a target goal by changing the body of text in one ormore of these structured stylistic feature categories.

Surface-level stylistic features 102 include stylistic elements ofcontent that are expressed at a surface level. For example, the surfacestylistic features 102 include the use of punctuation (commas,semicolons, colons, periods, question marks, exclamations, and dashes).The surface stylistic features 102 also include the total words in thesentence, the unique words in a sentence, the average number ofcharacters in a sentence, and the average word length.

For example, surface-level stylistic variations may include the use ofdifferent punctuation, including the use or non-use of Oxford commas. Inone example variation, the Oxford comma is used: There are prisons,offices, and colleges. In another example variation, the Oxford comma isnot used: There are prisons, offices and colleges. The use or non-use ofthe Oxford comma is quantified at the surface-level as discussed below.

The surface-level stylistic features 102 may be quantified into multiplefeatures. Each of the features may be numerically quantified. At thesurface-level, the number of occurrences of punctuation is quantifiedinto seven (7) features. The occurrences of punctuation featuresincludes commas, semicolons, colons, periods, questions marks,exclamations, and dashes. Other surface-level features including theaverage total words in a sentence, the average unique words, averagecharacters in a sentence, and average word length, are also quantified.These four (4) features in combination with the seven punctuationfeatures provide a total of eleven (11) surface-level stylistic features102 that may be quantified.

Lexical-level stylistic features 104 include stylistic elements ofcontent that are expressed by the used vocabulary words. That is, thelexical-level stylistic features 104 look at the particular words thatare used in a body of text. The lexical-level stylistic features 104include word choice and a characterization of the word as formal,informal, colloquial, literary, concrete, abstract, subjective, orobjective. The lexical-level stylistic features 104 also includeHapx-legomena and Dis-legomena, meaning the richness of vocabulary andusage of unfamiliar words.

For example, lexical-level variations may include the use of moreliterary words than colloquial words or the use of more abstract wordsthan concrete words. In the following example, the more colloquial andconcrete word “tasty” is used in one variation and the more literary andabstract word “palatable” is used in the other variation. In the firstvariation: Chocolates, although tasty, can kill you. In the othervariation: Chocolates, although palatable, can kill you.

The lexical-level stylistic features 104 may be quantified into multiplefeatures. At word-level, a total of eight (8) features may bequantified. For example, from the body of text being evaluated, thefraction of words of the body of text that are formal, informal,colloquial, literary, concrete, abstract, subjective, and objective arequantified. Additionally, hapax-legomena and dis-legomena words arequantified. That is, words that are considered to be hapax-legomena anddis-legomena, which are features that are indicative of the richness ofvocabulary in a given piece of text, are quantified as two (2) features.Thus, a total of ten (10) stylistic features are quantified at thelexical-level.

The syntactic-level stylistic features 106 include stylistic elements ofcontent that are expressed via the structuring of the sentences.Syntactic-level features 106 includes evaluating the type of sentencesas simple, complex, compound, and compound-complex. Syntactic-levelfeatures 106 also include evaluating sentences that have loose syntacticstructure and periodic syntactic structure, including the height andwidth of the syntactic parse.

For example, syntactic-level variations may include evaluating thesentences in a body of text for piled-up adjectives, detached adjectivalclause, and active-passive switch. For instance, in one syntactic-levelvariation, the sentence “I shall always remember my first visit toBoston” includes certain types of syntactic-level features. In anothersyntactic-level variation, the sentence “My first visit to Boston willalways be remembered” includes other types of syntactic-level features.

The syntactic-level stylistic features 106 may be quantified intomultiple features. At syntactic-level the fraction of sentences thathave simple, complex, compound-complex, and compound syntactic structureis quantified into fours (4) features. Apart from these, the“loose-ness” and “periodic-ness” of the sentences is quantified into two(2) features by computing the average height and width of syntacticparse trees. At the syntactic-level, a total of six (6) features arequantified.

These quantifications result in a 27-dimensional feature vector(henceforth represented as x) for a given piece of text. The featurevector may be referred to as a stylistic feature vector.

Given the structure and quantification of stylistic features illustratedin the chart 100 of FIG. 1, one objective is to model the effect ofthese stylistic features towards one or more target goals. The objectiveis met by training a de-confounded prediction model, which can then beused to understand which of the stylistic features “contribute” towardsthe one or more target goals in a causal manner. FIG. 2 is an examplechart 200, which shows the stylistic feature structure 100 from FIG. 1for a stylistic feature structure and a set of target goals. Asmentioned, a first objective is to determine which of the stylisticfeatures may have a causal influence on particular target goals.

Chart 200 illustrates multiple target goals. The target goals are goalsthat an author of a body of text desires to achieve. For example, onetarget goal for a body of text is shareability on social media. Thisrelates to the shareability of content on social media platforms, whichmay be a crucial aspect for content virality. Another target goal for abody of text may be search-engine optimization. This relates to thediscoverability of content by search engines and is crucial indetermining page visits. Another target goal for a body of text may bevalence. Valence relates to the pleasant or unpleasant nature of thetext that that reader reads. Another target goal for a body of text maybe arousal. Arousal relates to the intensity of the stimulus that thereader experience on reading a text. Another target goal for a body oftext may be dominance. Dominance relates to the concepts of control thatthe reader experiences on reading a text. Dominance includes theinfluence of psychosomatic factors in the body of text. Another targetgoal is reading time. Reading time relates to the time it takes to readthe text. Reading time is inversely proportional to the complexity ofthe body of text. Finally, another example target goal iscomprehensibility. Comprehensibility relates to the readers grasp andunderstand the text they are reading.

With the stylistic features and target goals in mind, the next step isto train a model that determines and models a causal relationshipbetween the stylistic features and the target goals to understand whichstylistic features have a causal influence on particular target goals.Once a model is trained, then the model may be used to generate andprovide causal stylistic feature prescriptions that informs the authoror creator of content how the content is going to perform based on oneor more target goals. The model may suggest changes to the content tobetter align the stylistic features with the forward-looking target goalfor the body of text.

A standard technique in machine learning to train a model is toformulate this as a regression task and use the variable-importance(e.g. (3-coefficients in regression models) to provide prescriptions.However, there is a major issue in providing the prescription using suchmethods: an author chooses to apply a style based on a given piece ofcontent, i.e., the style applied on a piece of content is not randomlychosen. Therefore, if one builds a supervised model that estimatestarget goals using the style aspects in a corpus without anyconsideration of the content, the estimate effect of the style on thetarget metrics will be confounded by the content of the text. In otherwords, it is possible for the model to attribute a content-imposedelement of style to a goal—which will be erroneous. The right approachis therefore to account for the joint effect of such confounders (theaspects of content leading to a style) and the style. However, theaspects of content that lead to a style are usually not observed. If themodel is estimated without accounting for the confounders, the size and,potentially, the direction of the effect of alternate styles on targetgoals will be wrongly estimated.

The technical solution for training the model as described belowimproves upon the standard technique in machine learning. The technicalsolution below provides an estimate of the confounders and accounts forthe confounders to get the correct effect of changes in the style ontarget strategy metrics. FIGS. 3 and 4 illustrates a technical solutionthat includes how components interact with each other to provideauthoring-time assistance.

FIG. 3 is a block diagram of an algorithm 300 to model target goals fromstylistic features. The algorithm 300 provides a process to train ade-confounded prediction model 310. For every target goal to be modeled,a set of documents 302 (e.g., a set of historic content pieces) with theassociated metric of interest. For example, the set of documents 302have a known outcome relative to one or more target goals. With thisknown set of documents and their achievement of particular target goals,process 300 is used to determine which stylistic features present in thedocuments 302 have a causal influence on the target goals. In thismanner, the de-confounded prediction model 310 can be trained.

In the documents 302 it is known what target goal or target goals thedocument achieved and the level of that achievement. For instance, itmay be known that a body of text in the documents 302 achieved a certaintarget goal. The body of text may be a social media post and include thenumber of time the social media post was shared as in indication of thelevel of shareability of the body of text. Another body of text mayinclude a metric for the length of time the reader spent viewing thebody of text as an indication of the reading time target goal.

The documents 302 are input to a stylistic feature model 304. Thestylistic feature model 304 is configured to identify stylistic featuresfrom the documents 302. In some implementations, the stylistic featuremodel 304 uses a logistic regression model to extract and identify thestylistic features that are present in the documents 302. To assist inthe identification and extraction of the stylistic features, a list oflexicons 306 may be used by the stylistic feature model 304. Thestylistic feature model 304 is used to identify a set of the stylisticfeatures from the documents 302, which can be further used to build thede-confounded prediction model 310.

For every content, the style features “x” are extracted. That is, foreach body of text in the documents 302, the stylistic features that canbe categorized into the features of surface-level, syntactic-level andlexical-level are extracted. The stylistic feature model 304 assigns anumerical value which indicates the presence of the stylistic feature ina body of text. The output of the stylistic feature model 304 is astylistic feature vector 308, which is a 27-dimensional vectorrepresentation of stylistic aspects in text x is used to predict thetarget goal y based on a logistic regression formulation. It is worthnoting that there is no notion of causality so far. This logisticalregression model is based on establishing correlation between thefeatures x_(i) and the target goal y.

Each quantization is one block in the stylistic feature vector. Thestylistic feature vector 308 is a representation of the stylisticfeatures for the documents 302.

In the next step, the stylistic feature vector 308 is input into theuntrained de-confounded prediction model 310. The next step will involveinferring latent confounders z which will enable establishment of causalrelationships between x_(i)∈ x and y. These confounders are hiddenreasons (if any) for an author's choice of style—e.g. a topic of“mergers and acquisitions” will force author to use objective words.Another potential confounder could be the topic of the text—itinfluences both the stylistic aspects x as well as the target strategyy. In order to truly prescribe goal-driven style suggestions, theseconfounding factors need to be accounted for and their effects need tobe removed in the modeling.

FIG. 4 is a block diagram of the de-confounded prediction model 310 fromFIG. 3. The process of de-confounding involves inferring and getting ridof such confounding variables—the variables that influence both x and y.It is essentially because in the presence of these confounding variablesone cannot correctly infer causations using the identified correlationsbetween x_(i)∈x and y. However, it's not possible to (a) enumerate and(b) model all the confounding variables. There are three sub-steps forthis particular step of our proposed method. First, in step 420, thehighly correlated features x_(i)∈ x are excluded and a probabilistic PCAfactor model is fit on a training subset of standardized x data. Toassess the factor model, predictive check 440 is performed on a held-outx and find that the replicated dataset generated from the probabilisticPCA model given the inferred latent variables, is nearly same as theoriginal dataset (p-value=0.393 for p(t(x_(n,heldout)^(rep))<t(x_(n,heldout)))—where t(·) denotes our test statistics).

If the check was successful, the substitute confounder (z) which wasinferred while fitting the factor model is valid, i.e. z is aconfounder. As mentioned above, in the context of text, the z couldrepresent various latent variables (like topics) that impact both style(x) and goal (y). Because of difficulty in observing and/or modelingthese latent confounding variables, the inferred correlations cannot betrivially extended to causations. Obtaining a valid substituteconfounder allows us to correct (460) for such unobserved latentconfounders, and consequentially infer causal relationships. To do thiscorrection (460), the substitute confounder is concatenated with thefeatures x⊕z and fit a logistic regression model on the obtained vectorsto predict the target variable y. The confounder-corrected logisticregression model is now the trained de-confounded prediction model 310,which is then used for further causal prescriptions.

These steps are repeated for each of the target goals that are ofinterest to the enterprise. The result of the above process is ade-confounded prediction model that has learned a function f(x) so thatf(x) gives the most accurate value y.

The output of the process 300 is a table 470 for each of the targetgoals, with an indication of the causal influence of each of thestylistic features on the target goal. The features with an “x” have nocausal influence on the particular target goal. The stylistic featureswith a “check mark” have a causal influence on the particular goal. Thetable also includes a directionality (+ and −) to indicate the directionof the causal influence on the target goal. For example, a “+” indicatesthat an increase in the feature will increase the target goal. On theother hand, a “−” indicates that an increase in the feature willdecrease the target goal. The number indicates the relative strength ofthe causal influence on the target goal.

Using the trained de-confounded prediction model, a new body of text canbe evaluated against target goals and changes to the body of text can besuggested by the trained de-confounded prediction model in a way thatdoes not change the topic itself. Instead, the model can suggest changesthat influence the way in which the topic is conveyed to the reader. Themodel de-confounds the topic from certain stylistic features on acertain target goal. The de-confounded prediction model learnssubstitute confounders in a way that separates out the effect of theconfounding variables from stylistic features. What remains are notcorrelations, but causations, because there are no confounders remainingin the model.

Referring to FIG. 5, a process 500 illustrates using the trainedde-confounded prediction model 510 to prescribe suggestions to a body oftext to align with desired target goals. Referring also to FIG. 6, aprocess 600 illustrates the difference modelling performed to determinethe specific prescriptions to suggest and output to a graphical userinterface (GUI). A body of text 502 is input into the stylistic featuremodel 504. In this example, the body of text 502 does not have a knownoutcome for any target goals. Instead, the process 500 evaluates thebody of text 502 to determine the current state of aligning with one ormore target goals and to make suggestions to alter the current alignmentwith the target goals.

The stylistic feature model 504 is based on a logistic regression model,which uses a list of lexicons 506 to identify all of the stylisticfeatures present in the body of text 502. The stylistic feature model504 outputs a stylistic feature vector 508, which is used to predict thetarget goals for the body of text 502. The stylistic feature vector 508is input into the trained de-confounded prediction model 510. Thetrained de-confounded prediction model 510 outputs a predicted value 512for a particular target goal. The value may or may not be the equal to auser-specified target value. Referring also to FIG. 6, a user may entera user-specified target value.

The trained de-confounded prediction model 510 performs a comparisonbetween the predicted value 602 and the user-specified target value 604.Depending on the magnitude and directionality of the difference (y-ŷ)between predicted target value ŷ and the user-specified target value y,use the inferred causal relationships to recommend stylistic changesthat can reduce the difference between ŷ and y.

For example, if there is no difference between the predicted value 602and the user-specified target value 604, then that ends the algorithmbecause no stylistic prescriptions are required 606.

For example, if there is difference where the predicted value is lessthan the target value 608, then the model identifies stylistic featureswith positive causal influence 610, such as feature 2 and feature 19 inthe table 612. Then, based on the magnitude of the causal influence forthe features 614, the stylistic prescriptions are ranked. In thisexample, the magnitude of feature 2 is larger than the magnitude offeature 19 so feature 2 is ranked higher than feature 19. The modeloutputs a suggestion 616 on a GUI to increase feature 2 (e.g., increasethe formal words) and to increase feature 19 (e.g., use more complexsentences).

If there is a difference where the target value is less than thepredicted value 618, then the model identifies the features with anegative causal influence 620, such as feature 5 as identified in table622. If there is more than one feature, then the features are rankedbased on the magnitude of the influence 624. In this example, the modeloutputs a suggestion 626 on a GUI to decrease feature 5 (e.g., decreasethe usage of subjective words).

The usefulness of our identified stylistic features is evaluated byadding them to state-of-the-art models for emotion predictions in astructured manner. Two standard datasets for emotion prediction areconsidered—The EmoBank dataset and The Facebooks posts dataset. In Table1, comparisons with multiple baselines illustrate that writing style isa compound factor of several stylistic elements and inclusion ofmulti-level stylistic features can lead to increased performance ondownstream tasks.

TABLE 1 Performance on the emotion prediction task. Multi-levelstylistic features ae concatenated with state-of-the-art baselines andcompare Pearson coefficient correlation. The EmoBank Dataset FB postsModels Valence Arousal Dominance Valence Arousal System [14] — — —  0.650   0.850 Ensemble (CLG) [15] ‡   0.635   0.375   0.277   0.727  0.355 +lexical +0.026 +0.003 +0.017* −0.010 +0.043 +surface −0.003*−0.005 −0.001 −0.006* +0.008 +syntactic +0.012 +0.006 +0.002* +0.004*+0.009 +all +0.039 +0.007 +0.020 +0.009 +0.062 Ensemble (CLG +   0.674  0.382   0.297   0.736   0.417 Our features) “+” denotes increase overbaseline ‡ due to addition of proposed stylistic features. Underlinedvalues are statistically significant with p < 0.001, while those with *are significant with p < 0.01.

A key consideration for the causal modeling is to miss out on keyaspects of the content contributing to the target goal in the pursuit ofconfounders. One way to evaluate this is to look at the modelingcapacity of all the variables against the confounder-corrected subset offeatures. For the ease of experimental modeling, a given target strategyis quantized into n bins—indicating, for instance, low, medium and high(i.e., 3 bins). This allows the problem to be modeled as an n-classclassification problem. The dataset comprises of over 10,000 pieces oftext annotated for VAD values. Table 2 shows the performance of thetrained logistic regression model, when taking the target strategy to bevalence (V), arousal (A) and dominance (D). Table 2 also shows theperformance of confounder corrected model—which replicates the vanillalogistic regression—suggesting that there is no loss of information inthe process of confounding.

Model Vanilla Logistic Regression Confounder Corrected Factor ModelEvaluation metric V A D V A D Precision 0.6113 0.5521 0.6946 0.61050.6503 0.5922 Recall 0.4884 0.5969 0.5525 0.4936 0.5911 0.5513 f1-score0.5461 0.6253 0.6112 0.5428 0.6219 0.6101 Was factor model good Yes YesYes as reconstructing (p-value = 0.38) (p-value = 0.43) (p-value = 0.37)

Next, a similar comparison is repeated with the models to predict theformality in a given corpus and report the results in Table 3. Thisinvolves changing the target variable y with formality scores, insteadof valence, arousal and dominance scores in Table 2.

TABLE 3 Prediction results of a causal logistic regression model forformality (target strategy). Vanilla Logistic Confounder CorrectedEvaluation metric Regression Factor Model Accuracy 0.6406 0.6388Precision 0.4912 0.4903 Recall 0.5957 0.5917 F1-score 0.5689 0.5641 Wasfactor model good at Yes reconstructing? * (p-value = 0.47) * Here,p-values are reported for the probability that reconstruction is notwithin specified bounds. This usage is not aligned with conventionalusage of p-values but serves the purpose of establishing the goodness offactor model in reconstruction.

Having established the predictive capabilities of proposed models, someof the stylistic elements that have causal significance with respect toa given target strategy are illustrated in Table 4. The target goalsunder consideration here are valence, arousal, dominance, and formality.While the first three are affect-related, formality is not. In Table 4,the mismatch is identified in terms of significance of stylisticfeatures as per conventional non-causal models and our proposed causalmodels.

Table 4 illustrates in red, the stylistic features that were identifiedas significant by non-causal modeling (i.e., by vanilla logisticregression model) and did not appear as causally significant duringcausal modeling (i.e., confounder corrected factor model). Similarly,the features in green are the ones that were identified as causallysignificant by causal-modeling and insignificant by non-causal modeling.The point under consideration is the mismatch between identifiedsignificant variables due to difference in modeling—for instance, whileuse of exclamation and subjective words is identified as significant forformality by conventional modeling, they are not done so by proposedcausal modeling. Furthermore, causal-modeling identifies additionalstylistic features, like use of complex-syntax, dashes, and periods ascausally significant for formality which are missed out by conventionalmodeling.

Table 4 also qualitatively reinforces the understanding of non-causalmodeling. Formality in text, which is an instantiation of targetvariable y, can be brought about due to the topic which is being wroteabout as well as the stylistic elements being used for conveying theinformation. The conventional model identifies subjective and objectivewords both as significant for predicting formality, which is incontradiction—subjective and objective words are two ends on the samespectrum; this can be attributed to the unobserved latent variables,like topic, which necessitates such usage. The technical solution of theapproached described in this document only identifies one of them, i.e.,objective words, as causally significant, and hence, there's nocontradiction—this can be attributed to de-confounding latent variables.

TABLE 4 Representative instances where some stylistic features mighthave a mismatch between causal and non-causal significance. Features inred denote mis-identified causal relationships due to a non-causalmodeling while the ones in green were left out (but correctly identifieddue to causal modeling). It's worth noting that the extent of influencealso varies. Valence Arousal Dominance Formality Causal Non-causalCausal Non-causal Causal Non-causal Causal Non-causal SignificanceSignificance Significance Significance Significance SignificanceSignificance Significance Colon Semicolon Subjective words Question markAbstract Period Period Exclamation Syntax Syntax Colon Colon Concretewords Concrete words Complex-syntax Subjective words Concrete wordsConcrete words Number of chars Number of chars Exclamation ExclamationDash Objective words Period Period Exclamation Exclamation Question markQuestion mark Abstract words Formal words Dash Dash Concrete wordsConcrete words Syntax Syntax Objective words Question mark Syntax SyntaxFormal words Period Period Question mark Dash Dash Informal wordsinformal words

Referring to FIG. 7, an example GUI 700 illustrates a body of text thathas been input into the algorithm 500 of FIG. 5. The trainedde-confounded prediction model 510 outputs stylistic featureprescriptions to better align the text along the target goal of arousal.For instance, in this example, the target goal is arousal. The body oftext is input into the algorithm 500 of FIG. 5. The output from thetrained de-confounded prediction model 510 output specific stylisticfeature prescriptions to make changes to increase the arousal of thedocument. For instance, the stylistic feature prescriptions are toincrease the number of exclamations mark, use more information words,and to use more subjective words. The body of text is highlighted withindicators (e.g., italics, bold, underline, colors, etc.) correspondingto each of the suggested changes. In some instance, specific words aresuggested to replace existing words in the body of the text to achievethe target goal.

An example GUI 710 also illustrates a body of text that has been inputinto the algorithm 500 of FIG. 5. The output from the trainedde-confounded prediction model 510 output specific stylistic featureprescriptions to make changes to increase the formality of the document.For instance, the stylistic feature prescriptions are to increase thenumber of dashes for conjunctions, use more formal words, and to usemore objective words. The body of text is highlighted with indicators(e.g., italics, bold, underline, colors, etc.) corresponding to each ofthe suggested changes. In some instance, specific words are suggested toreplace existing words in the body of the text to achieve the targetgoal.

FIG. 8 is a block diagram of a system 800 for generating stylisticfeature prescriptions to align a body of text with one or more targetgoals. The system 800 implements the algorithms 500 and 600 of FIGS. 5and 6, respectively.

The system 800 includes a computing device 802 having at least onememory 804, at least one processor 806 and at least one application 808.The computing device 802 may communicate with one or more othercomputing devices over a network 810. For instance, the computing device802 may communicate with a computing device 811 over the network 810.The computing device 802 may be implemented as a server, a desktopcomputer, a laptop computer, a mobile device such as a tablet device ormobile phone device, as well as other types of computing devices.Although a single computing device 802 is illustrated, the computingdevice 802 may be representative of multiple computing devices incommunication with one another, such as multiple servers incommunication with one another being utilized to perform its variousfunctions over a network.

The at least one processor 806 may represent two or more processors onthe computing device 802 executing in parallel and utilizingcorresponding instructions stored using the at least one memory 804. Theat least one processor 806 may include a graphics processing unit (GPU)and/or a central processing unit (CPU). The at least one memory 804represents a non-transitory computer-readable storage medium. Of course,similarly, the at least one memory 804 may represent one or moredifferent types of memory utilized by the computing device 802. Inaddition to storing instructions, which allow the at least one processor806 to implement the application 808 and its various components, the atleast one memory 804 may be used to store data, such as one or more ofthe stylistic feature prescriptions generated by the application 808 andits components used by the application 808.

The network 810 may be implemented as the Internet, but may assume otherdifferent configurations. For example, the network 810 may include awide area network (WAN), a local area network (LAN), a wireless network,an intranet, combinations of these networks, and other networks. Ofcourse, although the network 810 is illustrated as a single network, thenetwork 810 may be implemented as including multiple different networks.

The application 808 may be accessed directly by a user of the computingdevice 802. In other implementations, the application 808 may be runningon the computing device 802 as a component of a cloud network, where auser accesses the application 808 from another computing device over anetwork, such as the network 810. In one implementation, the application808 may be a word processing-type application, a comprehensive contentmanagement application such as, for example, Adobe Experience Manager,and/or a campaign management application such as, for example, AdobeCampaign. The application may include features that allow users create abody of text, select and set target goals for the body of text, andreceive automatically generated stylistic feature prescriptions to alignthe body of text with the desired target goals. The application may be astandalone application that runs on the computing device 802.Alternatively, the application may be an application that runs inanother application such as a browser application.

The application 808 includes a user interface 812, which includes anarea to enable the user to manipulate and edit a body of text. One ofthe many tools of the application 808 includes a stylistic featureprescription tool 814. The application 808 and the stylistic featureprescription tool 814 may implement the algorithm 500 of FIG. 5 and thealgorithm 600 of FIG. 6. The stylistic feature prescription tool 814takes as input a body of text via the user interface 812.

The application 808 includes a user interface 812 also referred tointerchangeably as a GUI 812. The user interface 812 is the front facinginterface with the application 808 that enables the user to manipulateand edit a body of text including using the stylistic featureprescription tool 814. The stylistic feature prescription tool 814 maybe initiated from the user interface 812.

More specifically, a stylistic feature model 820 receives the body oftext selected by the user via the GUI 812. The stylistic feature model820 identifies stylistics features from the body of text and populates astylistic feature vector with the stylistic features. A trainedde-confounded prediction model 822 receives the stylistic featurevector. The trained de-confounded prediction model 822 uses thestylistic feature vector and generates a prediction value for each ofthe one or more target goals. The trained de-confounded prediction model822 compares the prediction value for each of the one or more targetgoals to a target value for each of the one or more target goals. Then,the trained de-confounded prediction model 822 outputs one or morestylistic feature prescriptions to the body of text based on results ofthe comparing for display on the GUI 812.

In some implementations, the trained de-confounded prediction model 822includes a table for each of the one or more target goals, where thetable for each of the one or more target goals identifies the stylisticfeatures having a causal influence on the target goal for that table.The stylistic features may have no causal influence, a positive causalinfluence or a negative causal influence on a target goal.

Implementations of the various techniques described herein may beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. Implementations may beimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine-readablestorage device, for execution by, or to control the operation of, dataprocessing apparatus, e.g., a programmable processor, a computer, ormultiple computers. A computer program, such as the computer program(s)described above, can be written in any form of programming language,including compiled or interpreted languages, and can be deployed in anyform, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed on one computer or onmultiple computers at one site or distributed across multiple sites andinterconnected by a communication network.

Method steps may be performed by one or more programmable processorsexecuting a computer program to perform functions by operating on inputdata and generating output. Method steps also may be performed by, andan apparatus may be implemented as, special purpose logic circuitry,e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor for executing instructions and one or more memorydevices for storing instructions and data. Generally, a computer alsomay include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic, magneto-optical disks, or optical disks. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in special purposelogic circuitry.

To provide for interaction with a user, implementations may beimplemented on a computer having a display device, e.g., a cathode raytube (CRT) or liquid crystal display (LCD) monitor, for displayinginformation to the user and a keyboard and a pointing device, e.g., amouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

Implementations may be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation, or any combination of such back-end, middleware, orfront-end components. Components may be interconnected by any form ormedium of digital data communication, e.g., a communication network.Examples of communication networks include a local area network (LAN)and a wide area network (WAN), e.g., the Internet.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the scope of theembodiments.

What is claimed is:
 1. A computer-implemented method for generatingstylistic feature prescriptions to align a body of text with one or moretarget goals, the method comprising: receiving, at a trainedde-confounded prediction model, a stylistic feature vector, wherein thestylistic feature vector includes stylistic features from a body of textidentified by a stylistic feature model, the stylistic featuresincluding surface features, lexical features and syntactic features;generating, by the trained de-confounded prediction model using thestylistic feature vector, a prediction value for each of one or moretarget goals; comparing the prediction value for each of the one or moretarget goals to a target value for each of the one or more target goals;and outputting, for display on a graphical user interface (GUI), one ormore stylistic feature prescriptions to the body of text based onresults of the comparing.
 2. The method as in claim 1, wherein thetrained de-confounded prediction model includes a table for each of theone or more target goals, wherein the table for each of the one or moretarget goals identifies the stylistic features having a causal influenceon the target goal for that table.
 3. The method as in claim 2, whereinthe table for each of the one or more target goals identifies thestylistic features having a positive causal influence on the target goaland the stylistic features having a negative causal influence on thetarget goal.
 4. The method as in claim 3, wherein comparing theprediction value for each of the one or more target goals comprises:calculating a difference between the prediction value for each of theone or more target goals and the target value for each of the one ormore target goals; and if the difference is positive, decreasing orincreasing the stylistic features based on signs of correspondingweights for the stylistic features from the table for each of the one ormore target goals; and if the difference is negative, increasing ordecreasing the stylistic features based on signs of correspondingweights for the stylistic features from the table for each of the one ormore target goals.
 5. The method as in claim 2, wherein outputting, fordisplay on the GUI, the one or more stylistic feature prescriptionscomprises ranking the one or more stylistic feature prescriptions basedon a magnitude of the causal influence for the stylistic features on thetarget goals.
 6. The method as in claim 1, wherein outputting, fordisplay on the GUI, the one or more stylistic feature prescriptionscomprises identifying suggested changes to one or more words in the bodyof text.
 7. The method as in claim 1, wherein outputting, for display onthe GUI, the one or more stylistic feature prescriptions comprisesidentifying a new word to replace a word in the body of text.
 8. Themethod as in claim 1, wherein outputting, for display on the GUI, theone or more stylistic feature prescriptions comprises outputting, fordisplay on the GUI, a graphical indication of the one or more targetgoals.
 9. A system for generating stylistic feature prescriptions toalign a body of text with one or more target goals, the systemcomprising: at least one memory including instructions; and at least oneprocessor that is operably coupled to the at least one memory and thatis arranged and configured to execute instructions that, when executed,cause the at least one processor to implement an application, theapplication comprising: a graphical user interface (GUI) configured toindicate a selection of a body of text; a stylistic feature modelconfigured to receive the body of text, identify stylistic features fromthe body of text, and populate a stylistic feature vector with thestylistic features, the stylistic features including surface features,lexical features and syntactic features; and a trained de-confoundedprediction model that is configured to receive the stylistic featurevector, generate a prediction value for each of the one or more targetgoals using the stylistic feature vector, compare the prediction valuefor each of the one or more target goals to a target value for each ofthe one or more target goals, and output one or more stylistic featureprescriptions to the body of text based on results of the comparing,wherein the GUI is configured to display the one or more stylisticfeature prescriptions to the body of text.
 10. The system of claim 9,wherein the trained de-confounded prediction model includes a table foreach of the one or more target goals, wherein the table for each of theone or more target goals identifies the stylistic features having acausal influence on the target goal for that table.
 11. The system ofclaim 10, wherein the table for each of the one or more target goalsidentifies the stylistic features having a positive causal influence onthe target goal and the stylistic features having a negative causalinfluence on the target goal.
 12. The system of claim 11, wherein thetrained de-confounded prediction model is configured to: calculate adifference between the prediction value for each of the one or moretarget goals and the target value for each of the one or more targetgoals; if the difference is positive, decrease or increase the stylisticfeatures based on signs of corresponding weights for the stylisticfeatures from the table for each of the one or more target goals; and ifthe difference is negative, increase or decrease the stylistic featuresbased on signs of corresponding weights for the stylistic features fromthe table for each of the one or more target goals.
 13. The system ofclaim 10 wherein: the trained de-confounded prediction model isconfigured to rank the one or more stylistic feature prescriptions basedon a magnitude of the causal influence for the stylistic features on thetarget goals; and the GUI is configured to display the ranked stylisticfeature prescriptions.
 14. The system of claim 9, wherein: the trainedde-confounded prediction model is configured to identify suggestedchanges to one or more words in the body of text; and the GUI isconfigured to display the suggested changes to the one or more words inthe body of text.
 15. The system of claim 9, wherein: the trainedde-confounded prediction model is configured to identify a new word toreplace a word in the body of text; and the GUI is configured to displaythe new word to replace the word in the body of text.
 16. The system ofclaim 9, wherein outputting, for display on the GUI, the one or morestylistic feature prescriptions comprises outputting, for display on theGUI, a graphical indication of the one or more target goals.
 17. Thesystem of claim 9 wherein the trained de-confounded prediction modeluses a logistic regression model.
 18. A method of training ade-confounded prediction model, the method comprising: receivingtraining data including documents having known outcomes for a pluralityof target goals; generating, using a stylistic feature model, astylistic feature vector for each of the documents, wherein thestylistic feature vector includes stylistic features from a body oftext; generating, by a prediction model using the stylistic featurevector, a prediction value for each of the target goals; comparing theprediction value for the target goals to the known outcomes; andtraining the prediction model based on the comparison.
 19. The method ofclaim 18, wherein the trained de-confounded prediction model includes atable for each of the one or more target goals, wherein the table foreach of the one or more target goals identifies the stylistic featureshaving a causal influence on the target goal for that table.
 20. Themethod of claim 19, wherein the table for each of the one or more targetgoals identifies the stylistic features having a positive causalinfluence on the target goal and the stylistic features having anegative causal influence on the target goal.